Telegram Group & Telegram Channel
مقایسه زمانی BPE Tokenizer روی دو کتابخونه Hugging Face Tokenizers و OpenAI TikToken روی ولیدیشن دیتاست تاینی‌استوریز:

dataset = load_dataset("roneneldan/TinyStories")
texts = dataset["validation"]["text"]

# Load the GPT-2 tokenizer for both libraries
tiktokenizer = tiktoken.get_encoding("gpt2") # tiktoken
hf_tokenizer = Tokenizer.from_pretrained("gpt2") # Hugging Face tokenizers

# Measure tiktoken speed
start_time = time.time()
tiktoken_results = [tiktokenizer.encode(text) for text in texts]
tiktoken_time = time.time() - start_time

# Measure tokenizers speed
start_time = time.time()
hf_results = [hf_tokenizer.encode(text).ids for text in texts]
hf_time = time.time() - start_time

# Print results
print(f"tiktoken Time: {tiktoken_time:.4f} seconds")
print(f"tokenizers Time: {hf_time:.4f} seconds")

tiktoken Time: 2.6481 seconds
tokenizers Time: 16.7744 seconds



tg-me.com/pytorch_howsam/671
Create:
Last Update:

مقایسه زمانی BPE Tokenizer روی دو کتابخونه Hugging Face Tokenizers و OpenAI TikToken روی ولیدیشن دیتاست تاینی‌استوریز:

dataset = load_dataset("roneneldan/TinyStories")
texts = dataset["validation"]["text"]

# Load the GPT-2 tokenizer for both libraries
tiktokenizer = tiktoken.get_encoding("gpt2") # tiktoken
hf_tokenizer = Tokenizer.from_pretrained("gpt2") # Hugging Face tokenizers

# Measure tiktoken speed
start_time = time.time()
tiktoken_results = [tiktokenizer.encode(text) for text in texts]
tiktoken_time = time.time() - start_time

# Measure tokenizers speed
start_time = time.time()
hf_results = [hf_tokenizer.encode(text).ids for text in texts]
hf_time = time.time() - start_time

# Print results
print(f"tiktoken Time: {tiktoken_time:.4f} seconds")
print(f"tokenizers Time: {hf_time:.4f} seconds")

tiktoken Time: 2.6481 seconds
tokenizers Time: 16.7744 seconds

BY PyTorch Howsam


Warning: Undefined variable $i in /var/www/tg-me/post.php on line 283

Share with your friend now:
tg-me.com/pytorch_howsam/671

View MORE
Open in Telegram


PyTorch Howsam Telegram | DID YOU KNOW?

Date: |

Spiking bond yields driving sharp losses in tech stocks

A spike in interest rates since the start of the year has accelerated a rotation out of high-growth technology stocks and into value stocks poised to benefit from a reopening of the economy. The Nasdaq has fallen more than 10% over the past month as the Dow has soared to record highs, with a spike in the 10-year US Treasury yield acting as the main catalyst. It recently surged to a cycle high of more than 1.60% after starting the year below 1%. But according to Jim Paulsen, the Leuthold Group's chief investment strategist, rising interest rates do not represent a long-term threat to the stock market. Paulsen expects the 10-year yield to cross 2% by the end of the year. A spike in interest rates and its impact on the stock market depends on the economic backdrop, according to Paulsen. Rising interest rates amid a strengthening economy "may prove no challenge at all for stocks," Paulsen said.

How to Invest in Bitcoin?

Like a stock, you can buy and hold Bitcoin as an investment. You can even now do so in special retirement accounts called Bitcoin IRAs. No matter where you choose to hold your Bitcoin, people’s philosophies on how to invest it vary: Some buy and hold long term, some buy and aim to sell after a price rally, and others bet on its price decreasing. Bitcoin’s price over time has experienced big price swings, going as low as $5,165 and as high as $28,990 in 2020 alone. “I think in some places, people might be using Bitcoin to pay for things, but the truth is that it’s an asset that looks like it’s going to be increasing in value relatively quickly for some time,” Marquez says. “So why would you sell something that’s going to be worth so much more next year than it is today? The majority of people that hold it are long-term investors.”

PyTorch Howsam from id


Telegram PyTorch Howsam
FROM USA